Switch container networking from host to named bridge networks - #403

PostgreSQL, Valkey, Candlepin, Foreman, and Pulp move off host
networking onto dedicated bridge networks created by deploy_network:
foreman-db (internal PostgreSQL and its clients), foreman-cache
(Valkey and its clients), and foreman-app (Foreman, Candlepin, Pulp).
Services now reach each other by container DNS name (postgresql,
valkey, candlepin) instead of localhost/host.containers.internal.

PostgreSQL now also listens on a shared Unix socket
(postgresql_socket_dir) in addition to the bridge network, and the
postgresql/check_*/backup roles that ran administrative queries as the
postgres superuser switch to socket-based login rather than
password+host auth, since that socket is reachable from any container
sharing the mount without needing to open localhost access. The backup
role's own database dump connection info still needs a real reachable
host, so it picks the socket for internal databases and the configured
database_host otherwise via a new backup_database_host fact.

Candlepin gets an explicit candlepin_healthcheck_host/hostname pair and
a --resolve-based healthcheck so its container binds 0.0.0.0 while
still presenting the right SNI/Host identity used by its own cert
(added in a later commit). Foreman and Pulp's Redis/Candlepin URLs
move from localhost to their new container DNS names, driven by new
foreman_networks/foreman_migration_networks and
pulp_networks/pulp_migration_networks variables that also add the
foreman-proxy network to Foreman's rake/console containers when that
feature is enabled.

Co-authored-by: Cursor <cursoragent@cursor.com>

…n bridge network

foreman-proxy joins a new foreman-proxy bridge network (still also
published on host port 8443 via foreman_proxy_ports) rather than
running on host networking, while keeping its public identity
(foreman_proxy_name/foreman_proxy_url) unchanged: it still registers
with Foreman under its own real FQDN, so this is purely a network
plumbing change for how Foreman reaches the proxy container, not an
identity change. Podman's aardvark-dns needs a moment to reconverge
after the proxy container (re)starts before Foreman can resolve/reach
it again on the bridge, so a new shared wait_for_smart_proxy role
polls the proxy's /v2/features endpoint (from inside the foreman
container, using its client cert) with retries before anything tries
to register or use the proxy.

foreman_proxy's own tasks now restart the container as a plain task
(not a handler notify+flush) immediately before this readiness check
and the initial registration, since flushing handlers here would also
prematurely fire the paired "Refresh Foreman Proxy" handler before
registration has happened. The same readiness check runs again after
the role's normal end-of-role handler flush, since that flush restarts
the container again on effectively every deploy run. The backup role
reuses the same shared wait_for_reachable.yaml task after restarting
foreman.target, for the same reconvergence reason.

Co-authored-by: Cursor <cursoragent@cursor.com>

The IoP core network (iop-core-network) is now created through
deploy_network instead of calling containers.podman.podman_network
directly, so it picks up the same Netavark 2+ isolation-default
compatibility handling as the other bridge networks: it must not be
isolated from foreman-app/foreman-proxy, which it needs to reach.

The gateway container is dual-homed onto both iop-core-network and
foreman-app (still also published on host loopback at 24443), and
registers with Foreman using its container DNS name/port
(iop_core_gateway_registration_url) instead of localhost:24443,
waiting for that endpoint to become reachable from Foreman first
(mirroring foreman_proxy's own bridge-network readiness handling).
DB-consuming IoP services (advisor, inventory, remediation, vmaas,
vulnerability) join foreman-db alongside iop-core-network when the
database is internal, via a new iop_database_networks variable, and
iop_database_host switches from host.containers.internal to the
postgresql container DNS name in that mode.

iop_fdw's and iop_inventory's direct PostgreSQL admin queries (FDW
server/user-mapping setup, inventory schema/view creation) move from
password+host login to the same postgresql_socket_dir Unix socket the
other administrative roles now use, since containers sharing that
mount can reach it without depending on network reachability to the
database container.

Co-authored-by: Cursor <cursoragent@cursor.com>

Now that most services run on bridge networks instead of host
networking, containers need a way to resolve host-only name mappings
that operators define locally (e.g. in /etc/hosts) rather than through
real DNS. Podman's base_hosts_file setting copies the host's
/etc/hosts entries into each container's own /etc/hosts at container
start, which keeps those host-only mappings usable from the smart
proxy and other containers without reintroducing host networking.

Co-authored-by: Cursor <cursoragent@cursor.com>

Candlepin is now reachable from Foreman as https://candlepin:23443
rather than https://localhost:23443, so it needs a server certificate
valid for the candlepin hostname instead of sharing the localhost
certificate. certificates_hostnames gains candlepin, and
candlepin_tomcat_key/candlepin_tomcat_certificate now point at the
newly issued candlepin cert/key pair (candlepin_key/
candlepin_certificate in vars/certificates.yml) rather than the
localhost ones.

The IoP gateway keeps using the localhost certificate (it is still
reachable at both localhost and, once dual-homed onto foreman-app,
iop-core-gateway), so certificates now support a
certificates_hostname_aliases map of extra SAN names per issued
hostname. When the iop feature is enabled, base.yaml adds
iop-core-gateway as an alias for the localhost certificate so
Foreman's client-cert validation of the gateway succeeds over the
bridge network too.

Co-authored-by: Cursor <cursoragent@cursor.com>

foreman-installer answers migrated with an internal (db_manage)
database used to carry loopback hosts like localhost or 127.0.0.1,
which matched the installer's own host-networked PostgreSQL. Now that
internal PostgreSQL runs on the foreman-db bridge network as the
postgresql container, migrate_answers rewrites those loopback values
to postgresql for internal database_mode, leaving external database
hosts untouched.

Co-authored-by: Cursor <cursoragent@cursor.com>

Development deployments run the Rails process directly on the host,
so postgresql/valkey/candlepin/pulp keep network: host and their
localhost-based URLs there rather than adopting the new bridge
networks, avoiding a parallel set of development-only service URL
rewrites. remote-database.yaml, which stands up a standalone
PostgreSQL instance, needs the same explicit postgresql_network: host
override now that the role's own default changed to foreman-db.

Co-authored-by: Cursor <cursoragent@cursor.com>

Add a "Container networking" section to docs/developer/deployment.md
describing the four bridge networks, how services reach each other by
container DNS name, the foreman-proxy public/internal identity split,
why development deployments stay on host networking, and the
base_hosts_file DNS fallback. Update docs/iop.md's architecture,
service table, database, and certificate sections to reflect the
gateway's dual-homed networking and the internal database's bridge
access.

Co-authored-by: Cursor <cursoragent@cursor.com>

Adjust the remaining feature/integration tests to match services no
longer being reachable via host networking: PostgreSQL and Valkey are
no longer expected to have host-published ports (asserting instead
that the PostgreSQL Unix socket exists and that Valkey responds via
podman exec), Candlepin's status/TLS checks now run curl and openssl
s_client from inside the foreman container against the candlepin
container DNS name instead of curling localhost from the host, and the
webhook listener fixture targets host.containers.internal since
containers can no longer reach the host via localhost.

Remote execution tests now install the foreman-proxy container's SSH
key into the client's authorized_keys via a new
remote_execution_authorized_proxy_key fixture and verify the client is
DNS-resolvable from the proxy container, since the proxy's SSH-based
remote execution now runs from a bridge-networked container rather
than the host.

Co-authored-by: Cursor <cursoragent@cursor.com>